Hadoop MapReduce low latency matters! BGI Shenzen wins IDC Innovation Award

Last week IDC released the winners for the HPC Innovation Excellence Awards.

As per IDC’s announcement, BGI Shenzen saved millions of dollars while enabling faster processing of their large genome data sets. The IBM Platform Symphony product was used for the Hadoop MapReduce applications. Platform Symphony provides a low latency Hadoop MapReduce implementation. In conjunction with its unique shared memory management and the data aware low latency scheduler, it accelerates many life sciences applications by as much as 10 times over open source Apache Hadoop. 

Below is an excerpt from the release:

BGI Shenzhen (China). BGI has developed a set of distributed computing applications using a MapReduce framework to process large genome data sets on clusters. By applying advanced software technologies including HDFS, Lustre, GlusterFS, and the Platform Symphony MapReduce framework, the institute has saved more than $20 million to date. For some application workloads, BGI achieved a significant improvement in processing capabilities while enabling the reuse of storage, resulting in reduced infrastructure costs while delivering results in less than 10 minutes, versus the prior time of 2.5 hours. Some of the applications enabled through the MapReduce framework included: sequencing of 1% of the Human Genome for the International Human Genome Project; contributing 10% to the International Human HapMap Project; conducting research in combating SARS, and a German variant of the E. coli virus; and completely sequencing the rice genome, the silkworm potato genome, and the human gut metagenome. Project leader: Lin Fang

Congratulations to the BGI team for their well deserved recognition.

Rohit Valia
Director, Marketing
HPC Cloud and Analytics

The win in Monaco and a hot season continues



Mark Webber's winning dive into the pool, he's mid air!



As Red Bull Racing prepares for their upcoming race events, a recent highlight was at the Monaco Grand Prix where the team won the race for the third consecutive time! It was a nail biting race and that also resulted in a second consecutive win for Mark Webber in Monaco.

Platform Compuitng  logo on the car.  Very cool.

The partnership between Red Bull Racing and Platform Computing, an IBM company is an exciting one, seeing the team rise to the top and win the most prestigious race on the calendar… again was truly amazing! It was nothing but smiles on the Energy Station on race day. A big CONGRATULATIONS to the team and the best of luck as the season pushes ahead!

For more information on the partnership between Platform Computing and Red Bull Racing see

www.ibm.com/platformcomputing or check out the video or read the case study.

Next up the Grand Prix of Europe.

Big Data report from SC’11

In my previous blog I expressed high expectations for the Big Data-related activities at this year’s Supercomputing conference. Coming back from the show, I’d say the enthusiasm and knowledge on Big Data within the HPC community truly surprised me. Here are the major highlights from the show:

• Good flow of traffic at the Platform booth for Platform MapReduce. Many visitors stopped by our booth to learn more about Platform MapReduce – a distributed runtime engine for MapReduce applications. I found it easy to talk to the HPC crowd because many folks in this  community are already familiar with Platform LSF and Platform Symphony; both are flagship products from Platform that have been deployed and tested in large-scale distributed computing environment for many years. Since Platform MapReduce is built on similar core technology as what’s in those mature products, the HPC community quickly understood the key features and functions it brings to Big Data environments. Even though many users are still at  early stage of either developing MapReduce applications or looking into new programming models, they understand that a sophisticated workload scheduling engine and resource management tool will become critically important once they are ready to deploy their applications into production. Many HPC sites were also interested in exploring the potential of leveraging their existing infrastructure for processing data-intensive applications. For instance, questions on how a MPI and MapRedcue jobs can coexist on the same cluster were frequently asked at the show. The good news, Platform MapReduce is the only solution that can provide capability of supporting mixed workloads.

•  “Hadoop for Adults” -- This was a quote from one of the attendees after sitting through our breakfast briefing on overcoming MapReduce barriers. We LOVE it! The briefing lured over 130 people and well exceeded our expectations! Our presentation on how to overcome the major challenges in current Hadoop MapReduce implementations drew great interest. “Hadoop for Adults” sums up the distinct benefits Platform MapReduce brings. Platform Computing knows how to manage large-scale distributed computing environments. Bringing that same technology into Big Data environments is a natural extension for us. The reaction at SC’11 for Platform MapReduce was encouraging and a validation on our expertise in scheduling and managing workloads and overall infrastructure in a center.

• Growing momentum on application development. As sophisticated as always, the HPC community is at the forefront of developing applications to solve data-intensive problems across various industries and disciplines: cyber security, bioinformatics, electronic industry and financial services are just a few examples. Many Big Data related projects are being funded at HPC data centers and we are expecting a proliferation of applications coming out of those projects very soon.

The show is officially over but the excitement around Big Data will continue. For me, not only have I gained tremendous insights on the Big Data momentum in HPC, but I’m also pleased to see the overwhelming reaction for Platform MapReduce within the HPC community. Nothing beats pitching the right product to the right audience!  

Big Data’s Big Show at SC’11

It’s less than a week away, everyone in the HPC community are drumming up for SC’11. As someone who has been to SC for the past seven years, I was pleased to see that Big Data appears to be the  new kid on the block this year. Roughly 20 sessions will be dedicated to Big Data related topics at this year’s Supercomputing show. From basic training on Hadoop and MapReduce to the challenges and opportunities for exascale data analytics, we will hear wide range of discussions on  Big Data.  Platform Computing will also be hosting a breakfast briefing (free!) on how to overcome your MapReduce barriers in the morning of Wed., Nov 16 at the show. Registration details can be found here.

Traditionally, hot topics in HPC are often around performance, scaling, latency and bandwidth. It’s only been in the last couple of years that data intensive computing has become an area of interest in HPC, and it is heating up quickly! The reality is, now that hardware is getting faster and cheaper - thanks mainly to the advancement in processor technologies -- users can run more problems faster. As a result, more data is being generated and a lot of that data contains tremendous insights we could utilize to make better products and decisions. Sure, Big Data exists in Web 2.0, retail, telco companies as well as many other verticals in the enterprise space, yet there is no shortage of use cases in HPC. Areas such as cybersecurity, fraud detection, next-gen sequencing analysis are just a few such examples that fall into HPC arena – often applications in these space are both computationally and data intensive. 

HPC has long been considered the incubator for many bleeding edge innovations that will later trickle down and benefit mainstream applications. We believe Big Data is no exception. The annual Supercomputing show is always about showcasing leading edge science and technologies, Big Data certainly fits the bill.  I am looking forward to learning about new solutions for Big Data problems developed in HPC, as well as getting a better understanding of the specific requirements for this particular market. It will be an exciting week ahead, and we expect to hear some big buzz around Big Data at SC’11!

 

Stop by and visit Platform Computing at SC11 in booth #1117!

Blog Series – Five Challenges for Hadoop MapReduce in the Enterprise, Part 5

Challenge #5: Lack of Multiple Data Source Support

With this blog entry, we have reached the final chapter of the Hadoop challenge series. In this blog, I am going to discuss the fifth challenge for current Hadoop implementations – the lack of multiple data source support.

The current Hadoop implementation does not support multiple data sources; instead, it supports only one distributed file system at a time, the default being the Hadoop Distributed File System (HDFS). This restriction creates barriers for organizations whose data is stored in different file systems other than HDFS while implementing MapReduce applications in Hadoop.  For non-HDFS users, enabling MapReduce applications running in Hadoop environment means they have to first move their data from the current file system into HDFS, which can turn into a timing consuming and very expensive operation.  This limited capability for heterogeneous data support also leads to inferior performance and poor resource utilization due to an inflexible infrastructure.

The reality is, users want a platform that 1) supports various types of input data at the same time and outputs to their desired data sources (which could be different from the input type); 2) completes the   data conversion at runtime so no extra ETL steps are needed after the run.  (See the chart below for a high-level architecture layout for heterogeneous data support.) For instance, a user can send his input data stored in HDFS and output to a relational database such as Oracle and MySQL upon the completion of the run.  Such capabilities eliminate the data movement at both the beginning and the final stages of a MapReduce run, therefore, dramatically reducing the cost while improving the operation efficiency and driving   faster time to results. 

A high level diagram on heterogeneous data support

This capability of  heterogeneous data support in runtime can be considered as an alternative approach to traditional ETL function. The advantage of the former, while compared with the existing ETL tools, is that it provides a faster, cheaper and integrated new platform for users running Big Data applications.   

Having identified the 5 major challenges in the current Hadoop MapReduce implementation, we at Platform Computing has developed a solution – Platform MapReduce, an enterprise-class distributed runtime engine to not only address those barriers mentioned in this blog series, but also bring additional capabilities requested by users wanting to move their Big Data applications into production.  For detailed features and benefits delivered by Platform MapReduce, please visit: http://www.platform.com/mapreduce

Please join us at SC11 for a free breakfast briefing: “Overcoming Your MapReduce Barriers”.  Register today to secure your spot!

Blog Series – Five Challenges for Hadoop MapReduce in the Enterprise, Part 4

Challenge #4: Lack of Quality of Service

We are back after a short break.  The challenge with the current implementation of Hadoop MapReduce continues.  In this blog let’s take a look at the fourth challenge in the existing Hadoop stack – the lack of quality of service.

By high quality of service, we are referring to the capability of dynamically allocating available IT infrastructure based on workloads requirements, maximizing resource utilization and preventing silos. Those capabilities lead to better application performance and faster time to results, and therefore, provide high return on investment for the IT organization.  The current architecture design of the existing open source Hadoop stack puts limitations on the above capabilities. As mentioned in part 2 of this blog series, the single job tracker in the current Hadoop implementation is not separated from the resource manager, so as a result, the job tracker does not provide sufficient resource management functionalities to allow dynamic lending and borrowing of available IT resources. This creates a static IT environment in which each Hadoop application can only run on a pre-assigned set of resources at a given time and no exceptions are allowed. As the requirements of the application changes, resources will need to be re-configured manually to meet new demand. Such a static IT infrastructure brings the following issues for an IT organization:

·         Unable to provide the necessary and guaranteed services to multiple lines of businesses

·         Unable to manage real-time workload requirements

·         Slower performance and time to discovery

·         Increased management complexity

The result?  Underutilized resources and a higher total cost of ownership for IT.

In contrast to a static IT infrastructure, a sophisticated runtime built on a service oriented architecture (SOA) evolution.  brings quality of service to IT organizations committed to providing high quality services to their internal and/or external clients. Such a runtime solution will help transform IT into a true service provider and help meet demanding requirements (-high availability, dynamic resource allocations, ease of management, etc.) in the production environment. As new technologies such as Hadoop and MapReduce continue their penetration into the mainstream market, more applications will be developed and moved into production. Quality of service will undoubtedly become a critical consideration for IT in the next wave of the Big Data

Please join us at SC11 for a free breakfast briefing: “Overcoming Your MapReduce Barriers”.  Register today to secure your spot!

Why Combine Platform Computing with IBM?

You may have read the news that Platform Computing has signed a definitive agreement to be acquired by IBM and you may wonder why. I’d like to share with you our thinking at Platform and what our relevance is to you and the dramatic evolution of enterprise computing. Even though not an old man yet and usually too busy doing stuff, for once I will try to look at the past, present and future.

After the first two generations of IT architecture, centralized mainframe and networked client/server, IT has finally advanced to its third generation architecture of (true) distributed computing. An unlimited number of resource components, such as servers, storage devices and interconnects, are glued together by a layer of management software to form a logically centralized system – call it virtual mainframe, cluster, grid, cloud, or whatever you want. The users don’t really care where the “server” is, as long as they can access application services – probably over a wireless connection. Oh, they also want those services to be inexpensive and preferably on a pay-for-use basis. Like a master athlete making the most challenging routines look so easy, such a simple computing model actually calls for a sophisticated distributed computing architecture. Users’ priorities and budgets differ, apps’ resource demands fluctuate, and the types of hardware they need vary. So, the management software for such a system needs to be able to integrate whatever resources, morph them dynamically to fit each app’s needs, arbitrate amongst competing apps’ demands, and deliver IT as a service as cost effectively as possible. This idea gave birth to commodity clusters, enterprise grids, and now cloud. This has been the vision of Platform Computing since we began 19 years ago.

Just as client/server took 20 years to mature into the mainstream, clusters and grids have taken 20 years, and cloud for general business apps is still just emerging. Two areas have been leading the way: HPC/technical computing followed by Internet services. It’s no accident that Platform Computing was founded in 1992 by Jingwen Wang and I, two renegade Computer Science professors with no business experience or even interest. We wanted to translate ‘80s distributed operating systems research into cluster and grid management products. That’s when the lowly x86 servers were becoming powerful enough to do the big proprietary servers’ job, especially if a bunch of them banded together to form a cluster, and later on an enterprise grid with multiple clusters. One system for all apps, shared with order. Initially, we talked to all the major systems vendors to transfer university research results to them, but there was no taker. So, we decided to practice what we preached. We have been growing and profitable all these 19 years with no external funding. Using clusters and grids, we replaced a supercomputer at Pratt & Whitney to run 10 times more Boeing 777 jet engine simulations, and we supported CERN to process insane amounts of data looking for God’s Particle. While the propeller heads were having fun, enterprises in financial services, manufacturing, pharmaceuticals, oil & gas, electronics, and the entertainment industries turned to these low cost, massively parallel systems to design better products and devise more clever services. To make money, they compute. To out-compete, they out-compute.

The second area adopting clusters, grids and cloud, following HPC/technical computing, is Internet services. By the early 2000s, business at Amazon and Google was going gangbusters, yet they wanted a more cost effective and infinitely scalable system versus buying expensive proprietary systems. They developed their own management software to lash together x86 servers running Linux. They even developed their own middleware, such as MapReduce for processing massive amounts of “unstructured” data.

This brings us to the present day and the pending acquisition of Platform by IBM. Over the last 19 years, Platform has developed a set of distributed middleware and workload and resource management software to run apps on clusters and grids. To keep leading our customers forward, we have extended our software to private cloud management for more types of apps, including Web services, MapReduce and all kinds of analytics. We want to do for enterprises what Google did for themselves, by delivering the management software that glues together whatever hardware resources and applications these enterprises use for production. In other words, Google computing for the enterprise. Platform Computing.

So, it’s all about apps (or IaaS J). Old apps going distributed, new apps built as distributed. Platform’s 19 years of profitable growth has been fueled by delivering value to more and more types of apps for more and more customers. Platform has continued to invest in product innovation and customer services.

The foundation of this acquisition is the ever expanding technical computing market going mainstream. IDC has been tracking this technical computing systems market segment at $14B, or 20% of the overall systems market. It is growing at 8%/year, or twice the growth rate of servers overall. Both IDC and users also point out that the biggest bottleneck to wider adoption is the complexity of clusters and grids, and thus the escalating needs for middleware and management software to hide all the moving parts and just deliver IT as a service. You see, it’s well worth paying a little for management software to get the most out of your hardware. Platform has a single mission: to rapidly deliver effective distributed computing management software to the enterprise. On our own, especially in the early days when going was tough, we have been doing a pretty good job for some enterprises in some parts of the world. But, we are only 536 heroes. Combined with IBM, we can get to all the enterprises worldwide. We have helped our customers to run their businesses better, faster, cheaper. After 19 years, IBM convinced us that there can also be a “better, faster, cheaper” way to help more customers and to grow our business. As they say, it’s all about leverage and scale.

We all have to grow up, including the propeller heads. Some visionary users will continue to buy the pieces of hardware and software to lash together their own systems. Most enterprises expect to get whole systems ready to run their apps, but they don’t want to be tied down to proprietary systems and vendors. They want choices. Heterogeneity is the norm rather than exception. Platform’s management software delivers the capabilities they want while enabling their choices of hardware, OS and apps. 

IBM’s Systems and Technology Group wants to remain a systems business, not a hardware business nor a parts business. Therefore, IBM’s renewed emphasis is on systems software in its own right. IBM and Platform, the two complementary market leaders in technical computing systems and management software respectively, are coming together to provide overall market leadership and help customers to do more cost effective computing. In IBM speak, it’s smarter systems for smarter computing enabling a Smarter Planet. Not smarter people. Just normal people doing smarter things supported by smarter systems.

Now that I hopefully have you convinced that we at Platform are not nuts coming together with IBM, we hope to show you that Platform’s products and technologies have legs to go beyond clusters and grids. After all, HPC/technical computing has always been a fountainhead of new technology innovation feeding into the mainstream. Distributed computing as a new IT architecture is one such example. Our newer products for private cloud management, Platform ISF, and for unstructured data analytics, Platform MapReduce, are showing some early promise, even awards, followed by revenue. 

IBM expects Platform to operate as a coherent business unit within its Systems and Technology Group. We got some promises from folks at IBM. We will accelerate our investments and growth. We will deliver on our product roadmaps. We will continue to provide our industry-best support and services. We will work even harder to add value to our partners, including IBM’s competitors. We want to make new friends while keeping the old, for one is silver while the other is gold. We might even get to keep our brand name. After all, distributed computing needs a platform, and there is only one Platform Computing. We are an optimistic bunch. We want to deliver to you the best of both worlds – you know what I mean. Give us a chance to show you what we can do for you tomorrow. Our customers and partners have journeyed with Platform all these years and have not regretted it. We are grateful to them eternally.

So, with a pile of approvals, Platform Computing as a standalone company may come to an end, but the journey continues to clusters, grids, clouds, or whatever you want to call the future. The prelude is drawing to a close, and the symphony is about to start. We want you to join us at this show.

Thank you for listening.

Taming Big Data – A Recap of the O’Reilly Strata Conference

O’Reilly Strata, a conference dedicated to data science, held its second meeting of the year from September 22 – 23 in New York City.  The conference drew close to 500 attendees, including all the major technology providers in the space as well as user organizations from various industries that are dealing with Big Data problems. Platform Computing was one of the sponsors of the conference, and our introduction of the newly released Platform MapReduce 1.5 generated wide interest among conference participants.

Platform team greets booth visitors 

My takeaways from the conference are following:

1)   Data science is hot. The Strata conference lured participants from various industries. We met startups working on web analytics, energy companies trying to analyze their log files, banks of looking for use cases and solutions to help analyze their large data sets fast, and of course, Web 2.0 companies who are already at the forefront of tackling Big Data problems exploring  better solutions than what they are currently using. The enthusiasm around cracking Big Data   has never been higher. 

2)   Big Data market is still young. Many discussions we had at the show revealed that majority of the main stream organizations (excluding Web2.0 companies like Google, Yahoo, Facebook) are still at the early stages of the adoption where they are either exploring various technologies on the market, figuring out the proper applications built upon new technologies such as Hadoop and MapReduce, or in the midst of building small pilot projects. Comments such as “We are thinking about moving certain applications to Hadoop,” or “We only have one Hadoop project running at this time” were often heard at the conference.  A majority of the organizations that have large amounts of data are just beginning to tap into Big Data and looking into proper use cases.. The FAQ these days is “What to do with my data?” and there is no simple answer to that. 

3)   There’s a shortage of skilled resources. While Hadoop and MapReduce appear to be promising approaches to access and analyze Big Data, they are also new to developers, and the learning curve is rather steep. Adding to the length of the learning cycle is the fact that development tools in the ecosystem are still yet to mature. The reality is that there is lack of production quality applications for main stream user, most codes today are developed in-house and still being tested in R&D labs.  

4)   The market is fragmented. Various tools have been developed to fulfill the ecosystem while the mainstream market is still catching up on the basics.. We believe the gap will close and the market will eventually hit the tipping point as more applications become available. But for now, the path for Big Data remains wildly unpredictable. 

Big Data is here to stay. However, “Making Data Work”, the slogan at O’Reilley Strata conference, is no easy task. Companies dealing with  large amounts of data have a lot on their plates today --  disruptive technologies,  new application development,  understanding the meaning of the  new discoveries and their business impact, just to name a few. Needless to say, a well built-out ecosystem is critical to support all the efforts taking place in the market.    At Platform Computing, we are committed to not only providing the best solution in the ecosystem through leveraging our proven technology, but also working toward  bringing a viable,  end-to-end solution to the market.      

Blog Series – Five Challenges for Hadoop MapReduce in the Enterprise, Part 3

Challenge #3: Lack of Application Deployment Support

In my previous blog, I explored the shortcomings in resource management capabilities in the current open source Hadoop MapReduce runtime implementation. In this installment of the “Five Challenges for Hadoop MapReduce in the Enterprise” series, I’d like to take a different view on the existing open source implementation and examine the weaknesses in its application deployment capabilities. This is critically important because, at the end of day, it is the applications that a runtime engine needs to drive, without a sufficient support mechanism, a runtime engine will only have limited use.

 

To better illustrate the shortcomings in the current Hadoop implementation for its application support, we use below diagram to demonstrate how the current solution handles workloads.

As shown in the diagram, the current Hadoop implementation does not provide multiple workload support. Each cluster is dedicated to a single MapReduce application so if a user has multiple applications, s/he has to run them in serial on that same resource or buy another cluster for the additional application. This single-purpose resource implementation creates inefficiency, a siloed IT environment and management complexity (IT ends up managing multiple resources separately).

Our enterprise customers have told us they require  a runtime platform designed to support mixed workloads running across all resources simultaneously so that multiple lines of business can be served. Customers also need support for workloads that may have different characteristics or  are  written in different programming languages. For instance, some of those applications could be data intensive such as MapReduce applications written in Java, some could be CPU intensive such as Monte Carlo simulations which are often written in C++ -- a runtime engine must be designed to support both simultaneously.  In addition, the workload scheduling engine in this runtime has to be able to handle many levels of fair share scheduling priorities and also be capable of handling exceptions such as preemptive scheduling. It needs to be smart enough to detect resource utilization levels so it can reclaim functionalities when the resources are available.  Finally, a runtime platform needs to be application agnostic so that developers do not have to make code changes or recompile to adapt the runtime engine supporting their applications. The architecture design of the current Hadoop implementation simply does not provide those enterprise-class features required in a true production environment.   

Platform MapReduce v1.5 brings enhanced functionalities to MapReduce runtime

 

We are pleased to announce that Platform Computing 1.5 is now available.  Compared to its predecessor, Platform MapReduce v1.0 released in late June, the newly released version brings a number of enhancements in the key functionalities the product delivers.  Major improvements include the availability of the MapReduce application adapter technology, which allows users to execute their existing Hadoop applications without changing the code or recompile. Enhancements have also been made to the runtime layer so that mixed workloads can run on a same cluster simultaneously.  In addition, the support of IBM GPFS in the data layer is perhaps the most compelling capability in this release because it delivers a powerful solution to users running Hadoop applications on GPFS instead of a designated file system.

For more on Platform MapReduce 1.5, please see:  http://info.platform.com/rs/platform/images/Datasheet_PlatformMapReduce.pdf

So what does it all boil down to? Well, there are a number of immediate benefits with the new version:  

• The support of mixed workloads running on the same cluster simultaneously improves resource utilization and drives shared services model for IT, therefore multiple business lines can share the same infrastructure and a centralized IT management
• Increased developer productivities and choices. With the application adapter technology offered in Platform MapReduce 1.5, developers can build their applications using their preferred MapReduce programming framework and run their code without making changes to the code or recompile, thus  accelerating  the application development cycle while eliminating vendor lock-in.
• The integration of IBM GPFS and Platform MapReduce 1.5 allows users to run MapReduce applications directly on the data stored in GPFS instead of moving the data to a designated file system before the application execution, which can be a very costly operation.  In addition, the combined technologies deliver the best of both worlds to users running MapReduce applications.
• The unique capability of supporting different data input from output in Platform MapReduce 1.5 provides a more efficient approach to the ETL function as it eliminates the requirements for data staging at output, which can be a time consuming and expensive operation.

The past couple of months have been busy and exciting for us at Platform. We are seeing increased interest in the key functionalities offered in Platform MapReduce 1.5.  As Hadoop / MapReduce continue to gain market traction, users will become more educated on these emerging technologies and many will begin to move their MapReduce applications from labs to true production environments.  We believe Platform MapReduce will play a critical role in this transition by delivering a reliable, efficient and proven solution to users running MapReduce in production. 

Blog Series – Five Challenges for Hadoop MapReduce in the Enterprise, Part 2

Challenge #2: Current Hadoop MapReduce implementations lack flexibility and reliable resource management

As outlined in Part 1 of this series, here at Platform, we’ve identified five significant challenges that we believe are currently hindering Hadoop MapReduce adoption in enterprise environments.  The second challenge, addressed here, is a lack of flexibility and resource management provided by the open source solutions currently on the market.

Current Hadoop MapReduce implementations derived from open source are not equipped to address the dynamic resource allocation required by various applications.  They are also susceptible to single points of failure on HDFS NameNode, as well as on JobTracker. As mentioned in Part 1,  these shortcomings are due to the fundamental architectural design in the open source implementation in which the job tracker is not separated from the resource manager.  As IT continues its transformation from a cost center to a service-oriented organization, the need for an enterprise–class platform capable of providing services for multiple lines of business will rise.  In order to support MapReduce applications running in a robust production environment, a runtime engine offering dynamic resource management (such as borrowing and lending capabilities) is critical for helping IT deliver its services to multiple business units while meeting their service level agreements.  Dynamic resource allocation capabilities promise to not only yield extremely high resource utilization but also eliminate IT silos, therefore bringing tangible ROI to enterprise IT data centers.

Equally important is high reliability. An enterprise-class MapReduce implementation must be highly reliable so there are no single points of failure. Some may argue that the existing solution in Hadoop MapReduce has shown very low rates of failure and therefore reliability is not of high importance.  However, our experience and long history of working with enterprise-class customers has proved that in mission critical environments, the cost of one failure is measured in millions of dollars and is in no way justifiable for the organization. Eliminating single points of failure could significantly minimize the downtime risk for IT. For many organizations, that translates to faster time to results and higher profits.

Blog Series – Five Challenges for Hadoop MapReduce in the Enterprise

With the emergence of “big data” has come a number of new programming methodologies for collecting, processing and analyzing the large volume and often unstructured data. Although Hadoop MapReduce is one of the promising approaches for processing and organizing results from unstructured data, the engine running underneath MapReduce applications  is not yet enterprise ready. At Platform Computing, we have identified five major challenges in the current Hadoop MapReduce implementation:

·         Lack of performance and scalability

·         Lack of flexible and reliable resource management

·         Lack of application deployment support

·         Lack of quality of service

·         Lack of multiple data source support

I will be taking an in-depth look at each of the above challenges in this blog series. To finish, I will share our  vision on what an enterprise–class solution should be  that will not only address the five challenges customers are currently facing,  but also expand beyond those boundaries to explore the capabilities  of the next generation Hadoop MapReduce runtime engine.

 Challenge #1:  Lack of performance and scalability

Currently the open source Hadoop MapReduce programming model does not provide the performance and scalability needed for production environment, this is mainly due to its fundamental architectural design.   On the performance measure, to be most useful in a robust enterprise environment a MapReduce job should take  sub-millisecond to start,  but the job startup time in the current open source MapReduce implementation is measured in seconds. This high latency at the beginning can lead to subsequent delays in getting to the final results and cause significant financial loss to an organization. For instance, in capital markets of the financial service sector, a millisecond of delay can cost a firm millions of dollars.  On the scalability front, customers are looking for a runtime solution that is not only capable of  scaling one MapReduce application as the problem size grows,  but one that can also support multiple applications of different  kinds running across thousands of cores and servers at the same time.  The current Hadoop MapReduce implementation does not provide such capabilities. As a result, for each MapReduce job, a customer has to assign a dedicated cluster to run that particular application, one at a time.  This lack of scalability will not only introduce additional complexity into an already complex IT data center and make it hard to manage, but it also creates a siloed IT environment in which resources are poorly utilized.

A lack of guaranteed performance and scalability is just one of the roadblocks preventing enterprise customers from running MapReduce applications at production scale.  In the next blog,   we will discuss the shortcomings in resource management in the current Hadoop MapReduce offering and examine the impact it brings to organizations tackling “Big Data” problems.

Hadoop Summit 2011 Validates Platform MapReduce

“Big Data” is in and hot these days. This year’s Hadoop Summit 2011 attracted nearly 1,600 people, doubling the size of the conference from last year. Topics discussed at the Summit ranged from questions and concerns about Hortonworks, a fresh spinoff from Yahoo!, to various technical and use case discussions around Hadoop. While the center stage was dominated by full distribution players, such as Cloudera, Hortonworks and MapR, newcomers focused on providing alternative, best-of-breed component solutions in the stack are also emerging and getting increased traction from the market. This is not a surprise; the market for “Big Data” is still young and fragmented, and people at various phases of the technology adoption lifecycle are looking for solutions best suited for their needs. 

So the question is: full distribution or best-of-breed? 

At Platform Computing, we believe there is a need for both. For someone who is new to Hadoop and would like to experiment with this new programming model, a full distribution solution seems to be an easy way to get up to speed and get acquainted with Hadoop as it contains all the elements in the stack needed for running MapReduce applications. But for someone who is already Hadoop savvy and would like to bring their MapReduce applications into production, a whole new set of requirements will need to be met. Customers who need a production ready solution are seeking enterprise-class capabilities, such as 1) superior predictability of the infrastructure and distributed runtime engine for MapReduce jobs, so it meets the organization’s SLA requirements; 2) high resource utilization to eliminate a siloed environment while allowing organizations “do more with less”; 3) a rich set of management capabilities for operational efficiency; 4) high availability to ensure hardware and service failures do not require jobs to be manually recovered or restarted from scratch, and 5) of course, faster performance. 

The full distribution solutions currently on the market do not deliver those capabilities the mature market is looking for, that’s why Platform Computing is delivering Platform MapReduce, a best-of- breed, distributed runtime engine for MapReduce workloads, to fill in the gap. Launched on June 28, the eve of Hadoop Summit, Platform MapReduce received great traction at the event. As expected, users who have had a few years of experiences with either open source Hadoop or commercial solutions are well aware of the shortcomings in the existing options, and they were excited to hear about Platform MapReduce and the enterprise-class capabilities it provides. 

Built on Platform Computing’s decade’s worth of experience in managing and scheduling workloads in distributed environments, Platform MapReduce is designed using the same core technology that has powered many Fortune 1000 customers for their mission critical, most demanding workloads--bringing that capability to MapReduce environment is a natural market expansion for the company. Platform MapReduce addresses the major issues that are holding back the current market, and it is designed to help organizations overcome those barriers of moving MapReduce applications into production. The positive responses we’ve already received from the market are a solid validation of our solution, we are looking forward to bringing a new set of capabilities to the Hadoop world.

Platform MapReduce: Tackling Big Data, One Enterprise at a Time

“Big Data” seems to be on the tip of everyone’s tongue in recent months, and here at the Platform this has been no exception. Applying MapReduce applications to the data deluge has so much potential, but in Derrick Harris’ sage words “Hadoop may be hot, but it needs to be useful” (source: GigaOM).  With this in mind, Platform has set itself to applying its 18-year experience in policy-driven workload scheduling to the development of a MapReduce solution ready for the enterprise and to tackle its unique challenges. Back in March, Platform strongly hinted at a forthcoming product, but now it’s official.

Platform announced the launch of Platform MapReduce, the industry’s first enterprise-class, distributed runtime engine for MapReduce applications, with general availability to come at the end of July. The new solution will be able to manage MapReduce applications in a cluster (even multiple applications on a shared cluster) across an entire distributed file system. With more than 10,000 policy levels and support for up to 300,000 concurrent tasks, Platform MapReduce provides unparalleled manageability and scale, while ensuring high resource utilization to maximize ROI. Applicable to industries across multiple sectors, these key features can enable such diverse functions as compliance and regulatory reporting for financial services and government agencies; customer churn prevention for telecommunications; and genome sequencing analysis for life sciences.

Platform MapReduce also supports open distributed file system architecture, including immediate support for Hadoop Distributed File System (HDFS) and Appistry Cloud IQ – with more to come! To ensure that open source solutions, in this case those used with the Platform MapReduce distributed runtime engine, receive the world-class support enterprise customers demand, Platform has also signed the Apache Corporate Contributor License Agreement to contribute to the development of Apache-based, open-source Hadoop Distributed File System (HDFS).

With Platform MapReduce and world-class support, the enterprise is now ready to tackle the data deluge!